Add opt-in same-node scheduling preference for job and step pods#342
Open
jeanschmidt wants to merge 2 commits intoactions:mainfrom
Open
Add opt-in same-node scheduling preference for job and step pods#342jeanschmidt wants to merge 2 commits intoactions:mainfrom
jeanschmidt wants to merge 2 commits intoactions:mainfrom
Conversation
- Add ACTIONS_RUNNER_SAME_NODE_PREFERENCE env var to opt in to co-locating workflow pods on the same node as the runner pod - Inject a weighted (100) nodeAffinity preferredDuringScheduling rule matching the runner pod's kubernetes.io/hostname - Apply the preference in both createJobPod and createContainerStepPod - Add unit tests for useSameNodePreference and injectSameNodePreference Uses a soft preference (not a hard requirement) so pods still schedule if the runner's node has no capacity. Looks up the runner pod via ACTIONS_RUNNER_POD_NAME and logs a warning if the lookup fails. Signed-off-by: Jean Schmidt <[email protected]>
jeanschmidt
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in “same-node” scheduling preference for workflow-created pods in the packages/k8s hooks, so job pods and container step pods can be biased toward the same Kubernetes node as the runner pod (improving locality for caches/hostPath/NVMe, etc.) when explicitly enabled via env var.
Changes:
- Introduces
ACTIONS_RUNNER_SAME_NODE_PREFERENCEgating and utilities to inject a weight-100 preferred node affinity forkubernetes.io/hostname. - Looks up the runner pod’s
spec.nodeNameand applies the preference to job pods and container step pods (non-fatal on failure). - Adds unit tests covering the env gate and affinity injection behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| packages/k8s/src/k8s/utils.ts | Adds env constant + gate function and a helper to inject preferred node affinity into a PodSpec. |
| packages/k8s/src/k8s/index.ts | Adds runner-pod node lookup and applies same-node preference during job/step pod creation. |
| packages/k8s/tests/k8s-utils-test.ts | Adds tests for the env gate and affinity injection utility. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+49
to
+56
| const runnerPod = await k8sApi.readNamespacedPod({ | ||
| name: runnerPodName, | ||
| namespace: namespace() | ||
| }) | ||
| const runnerNodeName = runnerPod.spec?.nodeName | ||
| if (runnerNodeName) { | ||
| injectSameNodePreference(spec, runnerNodeName) | ||
| } |
Author
There was a problem hiding this comment.
False, container step pod is only created once.
Comment on lines
+205
to
+206
| await applySameNodePreference(appPod.spec) | ||
|
|
- Emit core.warning when ACTIONS_RUNNER_POD_NAME is unset but same-node preference is enabled - Add integration test verifying node affinity is applied to the job pod when same-node preference is enabled - Fix env var leak in useSameNodePreference unit tests by saving/restoring only the specific variable The silent early return made misconfiguration hard to diagnose. The warning surfaces it in Actions logs so operators can fix their runner pod setup. Signed-off-by: Jean Schmidt <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add opt-in same-node scheduling preference for job and step pods
Summary
Add a configurable same-node scheduling preference that biases job pods and container step pods toward the same Kubernetes node as their runner pod. Enabled via
ACTIONS_RUNNER_SAME_NODE_PREFERENCE=true. Disabled by default — no behavioral change unless explicitly opted in.Problem
When a runner pod creates a job pod (or a container step pod), the Kubernetes scheduler places it on any available node in the cluster. In setups where data locality matters — e.g. host-path caches, NVMe scratch volumes, pre-pulled images, or shared emptyDir volumes — this can cause:
execCpToPod/execCpFromPod) go over the network instead of staying node-localThere is currently no mechanism in the hooks to influence node placement. The only option is an external hook template (
ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE), but that's a static YAML file — it can't dynamically discover the runner pod's current node.Solution
Add a weight-100
preferredDuringSchedulingIgnoredDuringExecutionnode affinity to workflow pods, targeting the same node as the runner pod.How it works
ACTIONS_RUNNER_SAME_NODE_PREFERENCE=trueis set. If not, return immediately (no API calls, no overhead).ACTIONS_RUNNER_POD_NAMEto identify the runner pod.spec.nodeName.kubernetes.io/hostname= that node name.Why
preferredinstead ofrequiredA hard
nodeNamepin orrequiredDuringSchedulingwould cause job pods to fail scheduling entirely if the node has insufficient resources. The weight-100 preferred affinity gives maximum bias while remaining fault-tolerant — the scheduler treats it as a strong hint, not a hard constraint.Configuration
ACTIONS_RUNNER_SAME_NODE_PREFERENCE"true"to enable same-node preferenceACTIONS_RUNNER_POD_NAMERBAC requirement
The runner's ServiceAccount needs
getpermission on thepodsresource (already inrequiredPermissions) to look up the runner pod's node name. No additional RBAC is needed.Error handling
Non-fatal. If the runner pod lookup fails for any reason (permissions, pod not found, API timeout), a warning is logged via
core.warning()and the pod is created without the affinity — identical to the current behavior.What this does NOT change
ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATEFiles changed
packages/k8s/src/k8s/utils.ts— AddedENV_SAME_NODE_PREFERENCEconstant,useSameNodePreference()gate function, andinjectSameNodePreference()utility that builds the nodeAffinity entrypackages/k8s/src/k8s/index.ts— AddedapplySameNodePreference()function (env var gated, K8s API lookup, non-fatal), called from bothcreateJobPodandcreateContainerStepPodpackages/k8s/tests/k8s-utils-test.ts— 7 new tests covering the gate function and affinity injectionTest plan
npm run buildsucceeds (tsc + ncc)useSameNodePreferencereturnsfalsewhen env var is unset, empty,"false","TRUE", or"1"useSameNodePreferencereturnstrueonly when env var is exactly"true"injectSameNodePreferencecreates affinity from scratch on empty specinjectSameNodePreferenceappends to existing preferred scheduling entries (preserves template affinities)injectSameNodePreferencedoes not touchrequiredDuringSchedulingIgnoredDuringExecution